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Abstract 

We propose a theory of deterministic chaos for discrete systems, based on their 
representations in binary state spaces Q, homeomorphic to the space of symbolic 
dynamics. This formalism is applied to neural networks and cellular automata; it 
is found that such systems cannot be viewed as chaotic when one uses the Ham- 
ming distance as the metric for the space. On the other hand, neural networks 
with memory can in principle provide examples of discrete chaos; numerical sim- 
ulations show that the orbits on the attractor present topological transitivity 
and a dimensional phase space reduction. We compute this by extending the 
methodology of Grassberger and Procaccia to Q. As an example, we consider 
an asymmetric neural network model with memory which has an attractor of 
dimension D a = 2 for N = 49. 
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1. Introduction. 



Recently, discrete systems with a complex dynamical behavior have received 
a great deal of attention, for their relevance in fields ranging from theoretical 
biology to quantum gravity. For example, asymmetric neural networks t 1-4 ! 
can have a complicated dynamical behavior which is reminiscent of "chaos". 
Also cellular automata display ^ bifurcations between several possible dynamical 
regimes ^ , the most disordered of which has been described as "chaotic" ^ . Yet 
it is unclear precisely how this type of dynamics in discrete spaces is related to 
deterministic chaos in a Euclidean phase space. 

In this article we will examine how the definitions of deterministic chaos 
can be translated to the context of discrete state spaces. This will lead us to a 
formalism which we call "discrete chaos", that allows one to decide whether or 
not the complex dynamics of some finite systems can be viewed as chaotic in the 
limit in which the system grows to infinity. 

Unfortunately, for most finite systems there is no convenient quasi-represen- 
tation in terms of real variables. For example, in neural networks and cellular 
automata the relevant distance is the Hamming distance; this induces a discrete 
topology on the space of states that is distinct from the usual topology of IR n . 
There are different points of view on this problem, ranging from the fundamen- 
talist, which concludes that a finite system cannot be viewed as approximately 
chaotic, to the liberal, which reduces the definition of chaos to the exponential 
growth of the limit-cycle period with the size of the system. 

Our feeling is that chaos should not be limited to real variables, as these 
are idealizations of a reality which could be viewed equally well in terms of finite 
state spaces. Indeed, the fact that most real numbers have infinite algorithmic 
information ^ is not really satisfactory from a physicist's point of view. Yet 
some form of idealization is necessary to define chaos rigorously. 

Our purpose in this article is to propose a different idealization, inspired 
from symbolic dynamics t 9-13 !. We will assume that one is given a representation 
of the system through a sequence of iV-bit vectors. For example, one might 
consider the case where the different binary words carry information about the 
system at increasing temporal depths, e.g. by giving the iV-bit description of the 
system at every past tick of a clock. In general, the state of the system will be 
given by 

£ = {S(0),S(l),...,S(n),...}, 

where S (n) is a vector with components Si (n) = 0, 1, (i = 1, • • • , N). The set of 
such binary states will be denoted by O. 
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The approximation which makes this concept practical, akin to the 128-bit 
version of floating-point variables, is the truncation of the symbolic states to the 
first n words. This is a good approximation if the difference between states which 
coincide in the first n words belongs to a small neighborhood of the origin. We 
will formalize this demand through the assignment of a base for the topology on 
O, related to the cylinders of symbolic dynamics [ 10 > 13 l 

With this topology, the space O is homeomorphic to the one-sided shift 
space of symbolic dynamics. Our main contribution is to provide a definition 
of chaos for general dynamical maps in O. In symbolic dynamics one usually 
considers the shift map ex, which consists in erasing the word S(0) from the semi- 
infinite sequence and shifting the other slices by S(n) — > S(n — 1) I 10 l This 
example satisfies our definition of discrete chaos. But we stress that this is only 
one of many possible chaotic maps in O. 

We will consider functions which are continuous or discontinuous. Neural 
networks and cellular automata will turn out to be examples of discontinuous 
functions. For general discontinuous functions very little is known, basically due 
to the fact that analytically there is very little that one can prove. However, 
numerically one can distinguish several types of dynamical behavior. In IR"^ 
the Grassberger and Procaccia method is widely used to estimate the fractal 
dimension of attractors. We will extend its application to the space O in or- 
der to characterize different chaotic behaviors and define an effective attractor 
dimension. 

One important class of maps which we will consider in this paper corre- 
sponds to the case when the binary state represents the system at every past 
tick of a clock, as explained above. To define such a map one must provide a 
function which allows one to compute the new word S(0) from the state S. The 
left inverse of any such map is the shift map a of symbolic dynamics. Examples 
include neural network models and cellular automata. Note that in this case 
not all points of O represent possible histories: Instead, O plays the role of an 
embedding space for the attractor. 

The results of this paper can be generalized without difficulty to other 
alphabets besides the binary one, and also to the case where the space Q is the 
two-sided shift space I 11 ' 12 !, where a state is given by a sequence 

{•••S(-1),S(0),S(1),--.}. 

In this case our construction reduces to the invertible shift map when S(n) 
is taken to be the binary description of the system but once again we stress that 
this is only one of several possible dynamical maps F : O — > O. 

The organization of this paper is as follows. "Discrete chaos" will be defined 
in Sec. 2 and different types of dynamical maps in Q are discussed. In Sec. 3 we 
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will consider the correlation function C(p) and the Grassberger Procaccia method 
to compute the correlation dimension of the attractor. Numerical examples will 
be considered in Sec. 4 . In Sec. 5 we give the conclusion. 



2. Chaotic Dynamics of Binary Systems. 

Binary systems, like cellular automata and neural networks, are described, 
in general, by a set of N binary variables Si i = 1, . . . , N, or in short S, that 
evolve according to dynamical rules. The natural metric for these systems is the 
Hamming distance 

N 

d H (S-S') = J2\ s i- s 'i I • 

i=l 

The space {S} has 2 N possible states and so the topology constructed from 
dn is discrete. Generally one is interested in studying these dynamical systems 
in the limit N — > oo since that is where interesting statistical properties appear, 
such as phase transitions, and it is possible to use powerful techniques like mean 
field theory t 1-4 !. Furthermore numerical simulations which need to be done for 
finite, but large N, are understood as approximations of a system with infinite 
variables, much in the same way as floating point variables in computers are finite 
approximations of real numbers which generally have an infinite number of digits. 
Nevertheless for N — > oo, dn is no longer a distance and the topology is ill defined 
in that limit. That makes our understanding of binary systems quite different 
from that of dynamical systems in IR d or in differentiable manifolds where one 
works with the usual topology of the real numbers. Here we will overcome this 
situation by extending the phase space {S} to have an infinite number of states 
while preserving the equal status that the Hamming distance confers to each of 
the variables. That is to say, all the variables Si give the same contribution to 
the distance for any i. 

Let us consider the Cartesian product of infinite copies of {S} and call this 
space O. We denote the elements of O by 

5=(S(0),S(1),S(2),...). (1) 

We make O a topological space by introducing the following base: 

Af n (S) = {S' e fi|S' (to) = S (to) , Vm < n} , (2) 

with n = 1, 2, . . .. These base sets are closely related to the cylinders in one-sided 
shift spaces and O is homeomorphic to the space of symbols of the symbolic 
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dynamics with 2^ symbols I 10 ' 11 !. It follows that O is a cantor set. In symbolic 
dynamics the topology is usually derived from the metric 

d(S,S') = J2^d n (S-S'), (3) 

n=0 

where 

N 

d n (S-S') = J]|^(n)-^(n)|. (4) 

i=i 

is the Hamming distance of the n th copy of {S}. One can check that if S (m) = 
S' (m) Vm < n then d (S, S') < |^t, so that (2) and (3) define the same topology. 

Here and in the following our purpose is to study dynamical systems in 
Q generated by a function F : O — ► O. This function may be continuous or 
discontinuous, unless explicitly stated below. Allowing discontinuous functions 
in principle opens the door to a richer variety of systems, which include neural 
networks and cellular automata. 

We begin by generalizing in a natural way the definitions of chaos in subsets 
of IR^ (see for example Ref. [11]) to O. 

Definition 1: F has sensitive dependence on initial conditions on A C O if 3n E IN 
3 VS E A and VA/ m (S) 3S' E N m (S) n A and k E IN such that F k (S') £ 
M n {F k (S)). 

Definition 2: Let A C O be a closed invariant set. F : O — >■ O is topologically 
transitive on ^. C O if for any open sets U, V C 3n E 7L 3 F n (U) n F 7^ 0. In 
the last expression, if F is non invertible we understand F~ k {U) with /c > 0, as 
the set of all points S E O such that F^S) E U. 

Definition 3: Let A C O be a compact set. F : ^4 — ► is chaotic on if F has 
sensitive dependence on initial conditions and is topologically transitive on A. 

Definition 4- A closed subset M C O is called a trapping region if F (TV!) C M. 

Property 1: If F is a continuous function in O, F n (A^) is compact and closed 
Vn G IN. 

Proof. Since every closed subset of a compact set is compact, it follows that M. 
is compact and since F is continuous F n (A4) is compact. Since O is Hausdorff 
every compact subset of it is closed, so F n (M) is closed I 14 L ■ 

Definition 5: The map F : O — > O has an attractor if it admits an asymptotically 
stable transitive set, i.e., if there exists a trapping region Ai such that 

A = p| F n (M) 

n>0 
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and F is topologically transitive on A. 



Note carefully that the trapping region is defined in the O space while in the 
theory of dynamical systems in manifolds, it is defined in the manifold I 10-13 ' 15 !. 
This makes, most theorems (as those shoed in Ref. [15]) concerned with Cantor 
sets considered as attractors in manifolds to be not applicable. 

Property 2: If F is a continuous function in O, A is compact and closed. 

Proof: From property 1 if F is continuous, A is an intersection of closed sets, so 
it is closed. Since every closed subset of a compact space O is compact, it follows 
that A is compact. ■ 



Definition 6: A is called a chaotic attractor if F is chaotic on A. 

Lemma: Let F be a continuous function in O, if A is a chaotic attractor then it 
is perfect. 

Proof. By property 2, A is closed, it remains to prove that every point in A is 
an accumulation point of A. By contradiction, let So G A be an isolated point, 
then there exists n G IN 3 M n (So) H A = {So}. Then, by topological transitivity 
A has an isolated orbit (the orbit of S ) which implies that it is not sensitive to 
initial conditions on A. M 



Theorem: If F is a continuous function in O, and A is a chaotic attractor then it 
is a Cantor set. 

Proof: The theorem follows directly from property 2, the Lemma and the fact 
that a subset of a totally disconnected set is also totally disconnected. ■ 



In the following we will consider some examples of dynamical functions 
/ : O — > O. The first one is the one-side shift map a of symbolic dynamics 
which we introduce to familiarize the reader with the notation. 

i) The one-sided shift map a. 

The continuous map a defined by 

a(S(0),S(l),...) = (S(l),S(2),...), (5) 

is chaotic in O ' 10 L Note that a is non-invertible and its action loses the infor- 
mation carried by the binary state S (0). The meaning and usefulness of this 
map is quite clear in the context of symbolic dynamics when the Conley-Moser 
conditions are satisfied I 16 !. There one studies, in general, a non-invertible func- 
tion / : S — > S where S is a Cantor set embedded in JR N . The set S is divided 
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in 2^ sectors I a a = 0, 1, ...,2 . Then it is possible to establish a topological 
conjugation between / and a through a homeomorphism i/j, so that the following 
diagram commutes 




V I I </> • ( 6 ) 

Moreover, let £ = ip (x), then S (n) is the binary decomposition of the label a, 
such that f n (x) E I a . 

ii) Chaotic maps with non-trivial attractors in Q. 

The shift map can be modified to create maps which are homeomorphic 
to the shift map on an asymptotically stable transitive subset of the space of 
symbols. We introduce two very simple examples: 

Take the space of symbols O with N = 2, homeomorphic to S x S where S 
is the space of symbols with N = 1, that is the space of semi-infinite sequences 
£ = (So, Si, S2, ...)• Then consider the function / c :SxH->SxS given by 
f c = a x £. Where a is the usual shift function and ( is a right inverse of the 
shift function defined as follows: 

C (So, Si, S2, •••) = (0, So, Si, S2, •••) • 

It is easy to check that ( is a continuous function, and of course so is the shift: 
so f c is continuous. The set S x {0} is an asymptotically stable transitive set, 
on which the restriction of f c is the shift map a. 

As another example, consider the space O with N = 1. It can be split 
into the disjoint union of two Cantor sets O = Aq U Ai. Where Aq is the set of 
sequences such that So = and an analogous fashion for Ai. Take the continuous 
function f n = tt o a, where a is the shift map and tv projects O in Ao such that: 

7T (So, Si, S2, ...) = (0, Si, S2, •••) • 

Then the action of f n is given by, 

(So, Si, S2, •••) = (0, S2, S3, ...) . 

It is easy to check that A is a chaotic attractor of f^. 

in) Chaotic maps in O induced through chaotic maps in Cantor subsets of IR^. 

We will consider a homeomorphism which relates a Cantor set x C M, N to 
the space O and allows one to construct chaotic maps in O from chaotic maps in 



7 



X through topological conjugation. Let \ C IR be the Cantor set that results 
from taking the Cartesian product of N Cantor sets x%\ 

N 

X = ® Xi, 

i=i 

where the i th component Xi is constructed by suppressing from the interval [0, 1] 
the open middle 1/ai part, % = 1, . . . , N, cii > 1, and repeating this procedure 
iteratively with the sub-intervals, see Fig. 1. Now, we define 4> : O — > x by: 

oo 

& (S) = ( l n-l ~ In) Si (n - 1) (7) 

n=l 

where 

is the length of each of the remaining 2 n intervals at the n th step of the construc- 
tion of Xi- If ^ is endowed with the metric (3) and x C JR N with the standard 
Euclidean metric, is easy to show that is a homeomorphism. 

Now, if we have a map / : 1R N — ► IR^ which is chaotic in x we can 
construct a map F : O — > O which is chaotic in Q, and is defined through the 
commutation of the diagram 



^>T n- (9) 

This leads to an interesting practical application of the homeomorphism <f>, to 
realize computer simulations of chaotic systems on Cantor sets. If, for example, 
one iterates the logistic map / (x) = fix (1 — x) for [i > 4 with a floating-point 
variable, the truncation errors nudge the trajectory away from the Cantor set 
and eventually x — > — oo. The homeomorphism cf> suggests a natural solution to 
this, which is to iterate the truncated binary states rather than the floating-point 
variable. To iterate the dynamics, one computes Xi = (pi(S) Vz = 1, . . . , N by 
assuming that the truncated bits are all equal to zero, then applies / to obtain 
x' = f(x). Since x' generally does not belong to the Cantor set (because of 
truncation errors), in the process of constructing S' = ^ _1 (a; / ), at some n one 
will find that this point does not belong to either the interval corresponding to 
Si(n) — or to Si(n) = 1. This truncation error can be corrected by moving 
to the extremity of the interval which lies closest to x\. In this way, truncation 
errors are not allowed to draw the trajectory away from the Cantor set x C JR N . 
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iv) Binary systems with memory. 

Now we are going to define a map V : O — > O which is very useful to 
analyze binary systems with causal deterministic dynamics on N bits, such as 
neural networks, cellular automata, and neural networks with memory I 1-4 - 17 !. 
Let 

7 ,:O^{0,f}, (fO) 

% = 1, N, be a set of continuous or discontinuous functions. V : O — > O is 
then defined by: 

r i (s) = ( li (S),s i (o),s i (i),...). 

or in a short hand notation 

T(S) = ( 1 (S),S). (11) 
Such maps have the following properties. 

Property 3 The shift map (5) is a left inverse of V since from (11) a oT (S) = S. 
If Q has an attracting set A C O, then a is also a right inverse in the restriction 
of T to A, so that, T |X 1= o-. 

Proof: VS e A 35" e A such that T (S') = S. Since 

r (s>) = ( 7 (s f ) ,s') = s 

and 

S=(S(0),5i), 
where Si = (S(l), S(2), . . .), one sees that S' = S±. Thus, 

Foa(S) =T(Si) =T(S') = S. 

m 

Property 4 T has an attracting set A contained properly in Q. 

Proof. Given S there are 2^ states S' = (S'(0),S) of which only one, T(S) = 
('j(S), S), belongs to F(Q). Therefore the set 

a = p| r n (O) 

n>0 

is a proper subset of O. ■ 

Property 5 If V is continuous, then it is not sensitive to initial conditions. 
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Proof. T is a continuous map on a compact set, so it is uniformly continuous. 
Therefore there exists a 5 > such that for any S G O, d(S' , S") < 5 =>- 'j(S) = 
7(5") and hence d(T(S), T(S')) < 5/2, where the distance function is given by (3). 
Applying the same argument to each iterate T k (S) shows that d(T k (S), T k (S')) < 
5/2 k , which contradicts sensitivity to initial conditions. ■ 

Property 6 If V is continuous, then the attractor A is finite. 

Proof. From Property 4 we know that A exists. The property then follows from 
Property 5 above: Indeed, if V is not sensitive to initial conditions, then there is 
a n > such that \/S G O 

lim d(T k (S) -T k (S')) =0 

k— >oo 

VS" G Af n (S). The set i C O defined by 5 G A iff Vm > n, S(m) = 0, has 
a finite number of elements, namely 2 Nxn . The whole space O is the union of 
the n— neighborhoods of each element of A, and as we just showed the map V is 
contracting in each such neighborhood, so the number of points in the attractor 
cannot be greater than the number of elements of A, namely 2 Nxn . ■ 

Neural networks and cellular automata are binary dynamical systems in 
which the values of the state variables Si, i = 1, . . . , N, at time t depend on the 
state variables at time t — 1. These systems are described by a function T such 
that the functions 7$ depend only on the components S (0) . Therefore, all points 
S' G J\f n (S) for n > have the same evolution so that these systems are not 
sensitive to initial conditions. One can recover a very rough approximation of 
sensitive dependence on initial conditions by considering the growth of Hamming 
distance with time, rather than the metric (3) of symbolic dynamics. However, 
one cannot describe the behavior of these systems to be approximately chaotic: 
They are well known to have attractors that consist of a collection of periodic 
limit-cycles, and as we will see in Sec. 4, the points of these limit-cycles are scat- 
tered over configuration space without any effective lower-dimensional structure. 
In particular, given any one point on the attractor there is usually no other point 
"nearby" , even in the weak sense of the Hamming distance, that also belongs to 
the attractor. This fact makes most practical uses of chaos theory in prediction 
and control inapplicable. 

v) A compact topology for neural networks and cellular automata. 

Since neural networks and cellular automata in general are systems in which 
all the variables have the same type of interactions, it is natural to consider the 
Hamming distance as the metric (it is in fact the most widely used metric in the 
literature, see for instance Ref. [1-4] and the references therein). We have already 
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seen that the topological structure which the Hamming distance confers to the 
phase space does not conduce to chaotic behavior in the sense that we understand 
it even if we extend the phase space to O. However, not all the neural network 
and cellular automata models confer the same type of interactions to neurons, 
so the use the Hamming distance for the metric is not so compelling. The use 
of a different metric can lead to a completely different topology. The resulting 
system will in general display a very different dynamical behavior. For example 
the map x n+ i = ax n produces quite different dynamical behaviors for x n G IR 
and x n G S 1 . 

So, let us consider systems which evolve according to the rule 

R i (t+l) = f i (R(t)) (12) 
Ri = 0, 1; i = 1, M and take for the metric 

M 

d(S,S') = Y,^d n (S-S'). (13) 

n=0 

These systems include neural networks and cellular automata as particular ex- 
amples, but where the weight of the different neurons drops off as 2~ n . The 
metric (13) remains well defined in the limit M — > oo and once again we obtain 
the space O. In fact (12) and (13) with M — > oo are equivalent to (3) and (4) 
with N = 1 and Si (n) = R n . As we will see in the next section these systems 
can have a correlation dimension which is less than or equal to one. 



3. Correlation Function. 



In the theory of dynamical systems in IR^ one is interested in calculating 
the fractal dimension of the attractor in which the system evolves. To do so, 
following the method of Grassberger and Procaccia I 18 l one defines the correlation 
function C (p) as the average of the number of neighbors St, St', with S t = F l (S), 
which have a distance smaller than p. Since in IR^ the volume of a sphere of 
radius p grows like p N , one identifies the correlation dimension D a of the attractor 
with the growth rate in C (p) ~ p Da . This leads to the definition of the correlation 
dimension as 

/log(C(p))-log(C(p<))y 

p,„'-o V log M- log M ) y ' 

In order to have an analogous methodology to compute correlation dimensions 
in O, it is necessary to know how many states S' are within a distance less than 
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p from a given point S. Since O is homogeneous we can take S = 0. To do the 
calculation we make Q into a finite space by truncating the semi-infinite sequence 
to only T slices, and take the limit T — > oo in the end, that is: 

C (p) = Urn ^ £ (p - d (S, 0)) , (15) 

{S} 

where the distance is given by (3). Expressing Q(x) in terms of its Fourier 
transform u (k) = ttS (k) — ^ we have 

i i r+oo 

C (p) = lim - / dfc a, (fc) e**' £ e - ifcd < s -°> . 

- 7 " 00 {S} 

The sum over {S} can be evaluated easily obtaining 



^ e -*d(S,0) = 2 NT e -ikN /'pj cos - A_1 
|S| \n=0 / 



Using the identity s'mk/k = n^Lo cos 2^+ T we obtain the integral 

c w = s/.r* w( * ) ( !! r) W ^ 

which may be evaluated by standard complex variable methods, to obtain the 
final result for the correlation function in O, 

c w = 2^E(- 1 )*(fc)("- 2 *) N - < 16 > 

fc=o v y 

So we see that the scaling in O is not a power law as in JR N . However in the 
definition of the attractor dimension one is interested in calculating C (p) for 
p — > 0. For p < 2 equation (16) has the form 

C (P) = ^P N - ( 17 ) 

Therefore, the same techniques applied in IR^ can be used in O, in particular an 
effective "attractor dimension" will be given by (14). 
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4. Numerical Examples. 



We have examined numerically several of the binary systems which have 
been considered in the literature, including the random k = 4 cellular au- 
tomata I 7 1, and some neural network models such as those studied by Crisanti 
et. al. in the context of Shannon's entropy 

Random cellular automata of rank k consist of N binary variables Si = 0,1 
with the following dynamical rule. For each binary variable one chooses at ran- 
dom a boolean function /j : H\ — ► Z2 from k binary variables into one, among 
the 2 2 possible functions. Then, for each % = 1, . . . , AT, a set of k numbers 
{zi, 12, ■ ■ ■ , ik} is selected at random from {1, . . . , N}. These numbers are inter- 
preted as labels for the k inputs of the boolean function at that i. The evolution 
of the system in time is given by applying the boolean rules synchronously at the 
N variables: 

S i (t+l) = f i (S il (t),.-.,S ik (t)). (18) 

For k > 4 the system has very long cycles with periods of order e N and present 
the phenomenon of damage spreading which is the standard way in which a first 
"Lyapunov exponent" is assigned to such systems. 

In Ref. [2] Crisanti et. al. studied a binary neural network described by 
variables Si = ±1. The variables evolve in parallel according to the rule 

S l (t + l) = sgnr^J lJ S J (t)\ , (19) 



where 



T — T s 4- h T A 

J l3 ~ d ij ' fhJ ij 



is the synaptic matrix, with = jf- and Jj\ — —J{j being random independent 
gaussian variables with mean zero and variance a 2 = l/(N — 1)(1 + k). The 
parameter k measures the amount of asymmetry of the synapses. For k = 
the matrix is symmetric and the network has fixed points as attractors. For 
k > it is asymmetric and the network can have limit cycles as attractors. 
When k > k c = 0.5, long limit cycles are obtained, with period of order e N but 
with fluctuations of the same order. In Ref. [2] the Shannon entropy has been 
calculated numerically in the range k c < k < 0.9 and the scaling was found to 
be given by 

h~ (k-k c ) 1/2 . 

In the limit k — > 1, h attains a value which is very close to the maximum value 
log(2), characteristic of a random process. All of this indicates a high degree of 
complexity. 
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However, for both of the dynamical systems (18) and (19), the dynamics is 
not topologically transitive on the limit set of all periodic orbits: As we will see 
below, points on this set are isolated in Hamming distance, so most points do 
not even have any "near-neighbors" that might attempt to satisfy the conditions 
in the definition of topological transitivity. 

The number of returns to within a Hamming distance dn of an initial 
point on one of the long periodic orbits is given in the [Figs. 2, 3] for the cellular 
automata (18) with k = 4 and the neural network (19) with k = 1.2. We have also 
graphed the best-fitting Gaussian curve for comparison. The first obvious result is 
that there are no returns with dn < 50 in an automata with N = 200, in 5 x 10 6 
iterations of the dynamical map. The neural network was run with N = 100 
neurons for 5 x 10 5 iterations, and again no returns were found with dn < "j- 
Both the value of the nearest return and the fit to a Gaussian are consistent 
with a random process which produces patterns all over the configuration space 
without any restriction to a possible "attracting subspace" . This indicates a very 
high degree of algorithmic complexity ^ in the time-series, which reflects a lack 
of predictability, like Shannon entropy which is a statistical measure of disorder. 

Without anything analogous to a transitive "attractor" , none of the prac- 
tical applications of chaos theory can carry through for large values of N. The 
phase space reconstruction methods and other versions of the "method of ana- 
logues" I 19 ! fail because one finds no good analogue in any finite data set, for 
large N. The lack of close returns in binary systems can often be related to the 
failure to find an attractor on which the dynamics is topologically transitive. 

Another example, more in the spirit of the maps V is an asymmetric neural 
network with state-dependent synapses originally designed to recognize sequences 
of patterns, and described in Ref. [4] . As shown there, this system has a transition 
from a stable sequence reproduction to a disordered behavior. We shall modify 
the dynamical rule by introducing a memory in an analogous way as has been 
done in Ref. [17], as follows: 



Si(t + 1) = sgn 



T-l N 
n=0 7=1 



(20) 



where the synapses is given by 




(21) 



and 
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is the correlation of the state of the network with the pattern £ M . The patterns 
£^ = ±1 with \i = 1, . . . ,p + T are random independent, equiprobable variables, 
and p is a parameter of the model. The reader not familiar with the notation of 
Hopfield-type neural networks may refer to Ref. [4]. 

It is easy to show that the argument of the sign function referred to above 
as function 7^ does in fact vanish in O when T — > 00. So the map r : O — > O 
is discontinuous, and one cannot immediately rule out the possibility of non- 
periodic orbits [ 20 1. In practice however one always uses a finite memory in 
computer simulations, so the continuity is recovered and it is the sensitivity to 
initial conditions which is not valid. 

In applications it is often the case that two points can be considered to 
be distinguishable only if their mutual distance is greater than a small "cutoff" 
value A. If A > 1/2 T one can then claim that the map is "effectively" sensitive 
to initial conditions, which suggests that the ensuing dynamics may be correctly 
described as being "approximately chaotic" . This is best analyzed by computer 
simulation. 

We have run this system with T = 30, N = 49 neurons and p = 19 and 
find evidence for a non-trivial attractor with a low effective dimension: Unlike in 
the examples above there are substantially more near-returns than for a random 
sequence, as shown in [Fig. 4], and the correlation function C(p) produces an 
effective dimension D a ss 2 in the range 0.002 < p < 0.02 (see Fig. 5). 

For both the neural network (19) and the k = 4 cellular automata (18), 
we observe a very different behavior as expected; the correlation graph C(p) 
coincides for all p, with (16), within an accuracy of the 98% which means that 
D a PS N. That means that the orbits are scattered in the whole space O as one 
would expect from the fact that the distribution of returns to Hamming distance 
du is approximately Gaussian. 



5. Conclusion. 



From an initial ansatz, to replace the usual idealization of physical states as 
"points" on a differentiable manifold by another idealization as infinite "binary 
states" , we proceeded to define a topology which makes the truncation to finite 
states a valid approximation, in the same sense that the usual topology on IR^ 
allows one to approximate a real coordinate by a finite string of digits or bits. 
This lead us to a space O which is homeomorphic to the space of symbolic 
dynamics. 
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Continuous or discontinuous dynamical maps on the space of symbolic 
states can lead to attracting sets within Q, in which case an attractor is de- 
fined in the usual way. The dynamical map is said to be chaotic on the attractor 
if it is sensitive to initial conditions and topologically transitive. 

Finite systems such as neural networks and cellular automata without mem- 
ory that depend only on the previous time step and for which the different bits 
have comparable importance do not provide good approximations of chaos when 
the Hamming distance is used as the metric . The absence of even an approxi- 
mate manifestation of chaos has important practical consequences - for example 
we found that prediction models based on a search for analogous examples in a 
data set are not applicable because no good analogues are found in any reasonable 
amount of data. 

The practical value of the analysis of dynamical maps on the space O is 
probably limited to special complex systems problems where an extended binary 
description is more natural than a continuum description. Discrete space-time 
formulations of quantum gravity offer another potentially rewarding area of ap- 
plicability: There, the evidence for discrete small-scale structure combine with 
the perceived need of a space-time "sum over histories" interpretation lead to 
a formalism where one defines "states" to be truncated discrete space-time his- 
tories. An interesting example is the causal set formalism, where a partially 
ordered set or Poset is conjectured to constitute the minimal required structure 
to formulate a theory of quantum gravity. 

A priority in the continuation of this work is to further elucidate the chaotic 
properties of neural networks and cellular automata when a compact metric 
compatible with the topology of O is given. 
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Figure Captions 



[1] Construction of the Cantor sets Sj, i = 1, . . . , N by suppressing from [0, 1] 
the open middle 1/aj part, 1 < < oo. The remaining 2 n intervals at the 

n th step of the construction are of length l n = ^ (l — ^- J . 

[2] The number of times a /c = 4 random cellular automata with N = 200 
returns to a Hamming distance dn of a point half-way along the trajectory 
is represented as a function of dn (solid dots). The best-fitting Gaussian is 
also given for comparison (open dots). 

[3] The number of returns to Hamming distance du is shown, for an asymmet- 
ric neural network with N = 100 (solid dots). The best-fitting Gaussian is 
also given for comparison (open dots). 

[4] The number of returns to Hamming distance dn is given for a neural net- 
work model with memory, with N = 49. Unlike the previous examples, 
which correspond to dynamical systems without memory, we find many 
analogues. There is one return with du = 0; the system did not fall on a 
limit-cycle at that point because the dynamics also considers binary words 
further back in time. 

[5] The correlation graph N{p) gives the effective attractor dimension for the 
neural network with memory, D a « 2 in the range 0.002 < p < 0.02. The 
distance p is given by equation (3). 
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